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Preface 



It is with great pleasure that we present to you this tutorial volume entitled 
Validation of Stochastic Systems. It is one of the results of the Dutch-German bi- 
lateral cooperation project “Validation of Stochastic Systems” (VOSS), financed 
by NWO and DFG (the Dutch and German science foundations, respectively). 

In the early days of 2002, the idea emerged to organize a seminar at Schloss 
Dagstulrl, not the usual Dagstulrl seminar with primarily invited participants, 
but a seminar aimed at young(er) people, and for which the organizers assign 
themes to be worked upon and presented on. Following an open call announced 
via the Internet in the spring of 2002, we received many applications for partic- 
ipation. After a selection procedure, we decided to assign (mostly) teams of two 
researchers to work on specific topics, roughly divided into the following four 
theme areas: “Modelling of Stochastic Systems,” “Model Checking of Stochastic 
Systems,” “Representing Large State Spaces,” and “Deductive Verification of 
Stochastic Systems.” These are the titles of the four parts of this volume. 

The seminar was held in Schloss Dagstulrl during December 8-11, 2002 as part 
of the so-called GI/Research Seminar series. This series of seminars is financially 
supported by the Gesellschaft fur Informatik, the German Computer Society. At 
that point in time the papers had already undergone a first review round. Each 
of the tutorial papers was presented in a one-hour session, and on the basis of 
the presentations we decided to bring together a selection of them into a book. 
A second review round was performed throughout 2003; at the end of 2003 all 
contributions were finished. We are glad that Springer- Verlag was willing to 
publish it in their well-established Lecture Notes in Computer Science series, in 
particular in the “green cover” Tutorial subseries. 

To conclude this preface, we would like to thank NWO and DFG for making 
the VOSS bilateral cooperation project possible in the first place. Secondly, we 
would like to thank the Gesellschaft. fur Informatik for supporting the partici- 
pants of the seminar. We would like to thank the whole team at Schloss Dagstulrl 
for their willingness to host us and for their hospitality. We also thank the au- 
thors of the tutorial papers as well as the reviewers for their efforts; without 
you, there would not have been a workshop! Finally, we would like to thank 
Jose Martmez (of the University of Twente) for his work on the editing of this 
volume. 



Clrristel Baier 
Boudewijn Haverkort 
Holger Hermanns 
Joost-Pieter Katoen 
Markus Siegle 
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Abstract. We survey various notions of probabilistic automata and 
probabilistic bisimulation, accumulating in an expressiveness hierarchy of 
probabilistic system types. The aim of this paper is twofold: On the one 
hand it provides an overview of existing types of probabilistic systems 
and, on the other hand, it explains the relationship between these models. 
We overview probabilistic systems with discrete probabilities only. The 
expressiveness order used to built the hierarchy is defined via the exis- 
tence of mappings between the corresponding system types that preserve 
and reflect bisimilarity. Additionally, we discuss parallel composition for 
the presented types of systems, augmenting the map of probabilistic au- 
tomata with closedness under this compositional operator. 

Keywords: probabilistic automata (transition systems), probabilistic 
bisimulation, preservation and reflection of bisimulation, non-determinism, 
parallel composition. 



1 Introduction 

The notion of a state machine has proved useful in many modelling situations, 
amongst others, the area of validation of stochastic systems. In the literature up 
to now, a great variety of types of probabilistic automata has been proposed and 
many of these have been actually used for verification purposes. In this paper we 
discuss a number of probabilistic automata with discrete probability distribu- 
tions. For continuous-time probabilistic systems the interested reader is referred 
to [11, 33, 32, 17, 45, 4]. Models of stochastic systems that are not represented by 
transition systems can also be found in [22] and [70]. 

Due to the variety of proposed models it is often the case that results have 
to be interpreted from one type of systems to another. Therefore we compare 
the considered types of probabilistic automata in terms of their expressiveness. 
The comparison is achieved by placing a partial order on the classes of such au- 
tomata, where one class is less then another if each automaton in the class can 
be translated to an automaton of the other class such that translations both re- 
flect and preserve the respective notions of bisimilarity. Hence, bisimulation and 
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bisimilarity are central notions in this overview. Other comparison criteria are 
important as well, e.g. logical properties, logical characterization of bisimulation 
[61], complexity of algorithms for deciding bisimulation [9,13,31,80] and so on. 
We choose the comparison criterion formulated in terms of strong bisimulation 
because of its simplicity and because we work with transition labelled systems, 
for which bisimulation semantics arises naturally from the step-by-step behavior. 

A major distinction of probabilistic automata is that between fully probabilis- 
tic vs. non-deterministic ones. In a fully probabilistic automaton every choice is 
governed by a probability distribution (over set of states or states combined with 
actions). The probability distribution captures the uncertainty about the next 
state. If we abstract away from the actions in a fully probabilistic automaton, we 
are left with a discrete time Markov chain. Subsequently, standard techniques can 
be applied to analyze the resulting Markov chains. Sometimes, the incomplete 
knowledge about the system behavior can not be represented probabilistically. 
In these cases we should consider more than one transition possible. We speak in 
this case of a non-deterministic probabilistic automaton. Most of the models that 
we consider include some form of non-determinism and hence fall in the category 
of non-deterministic probabilistic automata. As pointed out by various authors, 
e.g. [47, 76, 3, 81] non-determinism is essential for modelling scheduling freedom, 
implementation freedom, the external environment and incomplete information. 
Furthermore, non-determinism is essential for the definition of an asynchronous 
parallel composition operator that allows interleaving. Often two kinds of non- 
deterministic choices are mentioned in the literature (see for e.g. [81]), external 
non-deterministic choices influenced by the environment, specified by having sev- 
eral transitions with different labels leaving from the same state, and internal 
non-determinism, exhibited by having several transitions with the same label 
leaving from a state. We use the term non-determinism for full non- determinism 
including both internal and external non-deterministic choices. 

We introduce several classes of automata, ranging from the simplest models 
to more complex ones. The questions that we will address for each individual 
class are: 

— the definition of the type of automaton and the respective notion of strong 
bisimulation; 

— the relation of the model with other models; 

— presence and form of non-determinism; 

— the notion of a product or parallel composition in the model. 

The set-up of the paper is as follows: Section 2 presents the necessary notions 
considering probability theory, automata (transition systems), and concurrency 
theory, in particular compositional operators. In section 3 we focus on the various 
definitions of probabilistic automata in isolation with their corresponding notions 
of bisimulation. In section 4 the operators of parallel composition are discussed. 
We address the interrelationship between the introduced types of automata in 
section 5. Section 6 wraps up with some conclusions. 
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2 Basic Ingredients 

2.1 Probability Distributions 

Let 17 be a set. A function p: i 7 — > [0, 1] is called a discrete probability distribu- 
tion, or distribution for short, on 17 if {x £ 17| p{x) > 0} is finite or countably 
infinite and J2 x& q p{x) = 1. The set {x € 17| p(x) > 0} is called the support- 
of p and is denoted by spt-(p). If x € 17, then p x denotes the unique probability 
distribution with p x (x) = 1, also known as the Dirac distribution for x. When 
p is a distribution on 17 we use the notation p[X\ for Y^i X px p(x) where X C 17. 
By V{fi) we denote the set of all discrete probability distributions on the set 
17. If p is a distribution with finite support {si,...,s n }, we sometimes write 
{si e- > n(s i), . . . , s n i— > p(s n )}. With this notation, p x = {x e-> 1}. 

Let p\ € D(S) and p 2 £ V(T). The product p\ x p 2 of pi and p 2 is a 
distribution on S x T defined by (p\ x p 2 )(s, t) = pi (s) ■ p 2 (t), for ( s , t) € S x T. 

If p € V(S x T), we use the notation p[s,T] for p[{s} x T\ and p[S, t] for 
p[S x {£}] . We adopt from [51] the lifting of a relation between two sets to a 
relation between distributions on these sets. 

Definition 1. Let R C S x T be a relation between the sets S and T. Let 
p £ D(S) and p! £ V{T) be distributions. Define p =r p! if and only if there 
exists a distribution v £ V{S x T) such that 

1. v[s,T] = p(s) for any s £ S 

2. v[S, t] = p'\t) for any t £ T 

3. u(s, t) ^ 0 if and only if ( s , t) € R. 

The lifting of a relation R preserves the characteristic properties of preorders 
and equivalences (cf. [52]). For the special case of an equivalence relation there 
is a simpler way to define the lifting (cf. [52, 81, 9]). 

Proposition 1. Let R be an equivalence relation on the set S and let p,p' £ 
D(S). Then p =r p! if and only if p[C } = p'[C \ for all equivalence classes 
C £ S/R. □ 

Lifting of an equivalence relation on a set S to a relation =r^a on the set 
T>(A x S ), for a fixed set A , will also be needed. 

Definition 2. Let R be an equivalence relation on a set S, A a set, and let 
p,p! £ V(A x S). Define 

P=r^ap' VC £ S/R,\/a £ A: p[a,C] = p'[a,C] 
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2.2 Non-probabilistic Automata, Markov Chains, Bisimilarity 

Throughout the paper we will use the terms automaton, transition system or 
just system as synonyms. 

Non-probabilistic Automata 

Definition 3. A transition system, TS for short, is a pair ( S , a) where 

1. S is a set of states 

2. a : S — * V(S) is a transition function, where V denotes the powerset of S. 

If (S,a) is a transition system such that s,s' € S and s' £ a(s) we write 
s — > s' and call it a transition. 

Often in the literature a TS is given as a triple, including besides the set of 
states and the transition function also a subset of initial states, or a single initial 
state. In this paper we will consider no initial states and therefore they are not 
present in the definition. Instead of a transition function one could equivalently 
consider a transition relation as a subset of S x S. Our choice here is to always 
present the transitions via a transition function. 

A way of representing a TS is via its transition diagram. For example, the sys- 
tem (S, a) where S = (si, S2, S3, S4} and cc(si) = {S2, S3}, a(s2) = {S4}, a(s3) = 
0(54) = 0, is represented as follows: 




The states S3 and S4 are terminating states, with no outgoing transitions. 

It is often of use to model the phenomenon that a change of state in a system 
happens as a result of executing an action. Therefore, labelled transition systems 
evolve from transition systems. There are two ways to incorporate labels in a 
TS: by labelling the states (usually with some values of variables, or a set of 
propositions true in a state) , or by explicitly labelling the transitions with actions 
or action names. In this paper we focus on transition labelled systems. 

Definition 4. A labelled transition system (LTS) (or a non- deterministic au- 
tomaton) is a triple (S,A,a) where 

1. S is a set of states 

2. A is a set of actions 

3. a : S — > V(A x S) is a transition function. 



When (S,A,a) is a LTS, then the transition function a can equivalently be 
considered as a function from S to V{S) A , the collection of functions from A 
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to V(S). As in the case of TSs, for any state s £ 5 of a LTS, every element 
(a, s') £ a(s) determines a transition which is denoted by s s'. 

The class of non-deterministic automata (LTSs) is denoted by NA. Deter- 
ministic automata, given by the next definition, form a subclass of NA. 

Definition 5. A deterministic automaton is a triple (5, A, a) where 

1. S is a set of states 

2. A is a set of actions 

3. a : S — > (S + 1) A is a transition function. 

Notation 1 We denote by + the disjoint union of two sets. The set 1 is a 
singleton set containing the special element *, i.e. 1 = {*}■ We assume that 
* ^ S. The notation ( S + 1) A stands for the collection of all functions from A 

to 5+1. 

The special set 1 and the disjoint union construction allow us to write par- 
tial functions as functions. Hence, in a deterministic automaton each state s is 
assigned a partial function a(s) : A — > 5+1 from the set of actions to the set of 
states, meaning that whenever a(s)(a) = s' for some s' £ 5, i.e. a(s) yf *, then 
there is a transition s s' enabled in 5. We denote the class of all deterministic 
automata by DA. 

We note that the class of automata DA exhibits external non-determinism, 
while in NA there is full non-determinism. 



• • 




• • • • • 

external non-determinism full non-determinism 



Markov Chains. The simplest class of fully probabilistic automata is the class 
of discrete time Markov chains. The theory of Markov chains is rich and huge 
(see, e.g., [57,48, 16,43]) and we only provide a simple definition of a discrete 
time Markov chain here. 



Definition 6. A Markov chain is a pair ( S,a ) where 

1. S is a set of states 

2. a : S — » V(S) is a transition function. 

Markov chains evolve from transition systems, when probability is added to 
each transition such that for any state the sum of the probabilities of all outgoing 
transitions equals 1. The class of all Markov chains is denoted by MC. If s £ S 
and a(s) = p with p(s') = p> 0 then the Markov chain (S,a) is said to go from 
a state s with probability p to a state s'. Notation: s p and s s'. 



Example 1. 



S — {so, Si, S2} 

«( s o) = {+) 0,Si i,s 2 i-> 1} 
«0i) = hl 0 
«(S2) = 



1 
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Bisimulation and Bisimilarity. Different semantics or notions of behavior 
can be given to labelled transition systems. We work with the bisimulation se- 
mantics (Milner [65, 66]) stating that two states in a system represented by LTSs 
are equivalent whenever there exists a bisimulation relation that relates them. 
A bisimulation relation compares the one-step behavior of two states and has a 
nice extension to the probabilistic case (as explored in [61]). In [54] probabilistic 
extensions of a number of other well known process equivalences have been stud- 
ied like probability trace, completed trace, failure and ready equivalence. Other 
probabilistic process equivalences are probabilistic simulation and bisimulation 
by Segala and Lynch [78, 76], Yi and Larsen’s testing equivalence [88], and CSP 
equivalences of Morgan et al. [67], Lowe [59] and Seidel [77]. An overview of 
several probabilistic process equivalences can be found in [58]. 

Definition 7. Let (S, A, a) and ( T , A, a) be two LTSs. A relation R C S x T 

is a bisimulation relation if for all ( s,t ) £ R and all a £ A the following holds 

if s -A s' then there exists t' £ T such that t -A t' and ( s',t ') £ R, and 

ift—*lf then there exists s' £ S such that s -A s' and ( s',t ') € R. 

Let s £ S and t £ T. The states s and t are called bisimilar, denoted by s ss t 
if there exists a bisimulation relation R with ( s,t } £ R. 

Example 2. For the following LTSs we have, for example, sq « £ 0 since R = 
{(s 0 ,to), (so, <2), (si,ti), (si,f 3 )} is a bisimulation. 




•«1 W * t2 



Remark 1. Instead of comparing states in two systems (S, A , a) and (T, A, (3) we 
can always consider one joined system ( S + T, A, 7) with y(s) = a(s) for s £ S 
and 7 (t) = /3(t) for t £ T. Therefore bisimulation can be defined as a relation on 
the set of states of a system. Furthermore, if R C S x S is a bisimulation, then it 
is reflexive and symmetric, and the transitive closure of R is also a bisimulation. 
Hence bisimilarity « is not affected by the choice of defining bisimulation as an 
equivalence. 

Definition 8. An equivalence relation R on a set of states S of a LTS is an 
equivalence bisimulation if for all ( s,t ) £ R and all a £ A 

if s A s' then 3 1' £ S: tAf, (s', t') £ R 

The states s and t are called bisimilar, denoted by s ss e t if there exists an 
equivalence bisimulation R with ( s , t) £ R. 
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By Remark 1, the following proposition holds. 



Proposition 2. Let (S,A,a) and (T,A,(3) be two LTSs, and let s £ S, t £ T. 
Then s « t if and only if s « e t. 

□ 

Bisimulation on DA is defined exactly the same as for NA i.e. with Defini- 
tion 8. 

The standard notion of probabilistic bisimulation is the one introduced by 
Larsen and Skou [61] originally formulated for reactive systems (see next sub- 
section). An early reference to probabilistic bisimulation can be found in [23]. 
In the case of Markov chains, bisimulation corresponds to ordinary lumpabil- 
ity of Markov chains [57,44,27]. In [86,85] it is shown that the concrete notion 
of bismulation for Markov-chains coincides with a general coalgebraic notion of 
bisimulation [68,53,74,64]. 

The idea behind probabilistic bisimulation is as follows. Since bisimilar states 
are considered “the same”, it does not matter which element within a bisimu- 
lation class is reached. Hence, a bisimulation relation should compare the prob- 
ability to reach an equivalence class and not the probability to reach a single 
state. In order to define bisimulation for Markov chains the lifting of a relation 
on a state S' to a relation on 'D(S), as defined in Definition 1 and explained with 
Proposition 1, is used. Note that the comments of Remark 1 are in place here 
as well. 



Definition 9. An equivalence relation R on a set of states S of a Markov chain 
(S, a) is a bisimulation if and only if for all (s,t) £ R 

if s /i then there is a transition t fj with, /u =r [A . 

The states s and t are called bisimilar, denoted by s « t, if there exists a bisim- 
ulation R with ( s , t) £ R. 



Definition 9 will be used, with some variations, for defining bisimulation 
relations for all types of probabilistic automata that we consider in this overview. 
However, note that in the case of Markov chains any two states of any two 
Markov chains are bisimilar, according to the given definition, since V = S x S 
is a bisimulation on the state set of any Markov chain (S, a). Namely, let ( S , a) be 
a Markov chain and s,t £ S, such that a(s) = /./, aft) = //, i.e., s ^ ^ 

Then for the only equivalence class of V, S, we have /./[S'] = 1 = //[S'] i.e. 
// =r // which makes s « t. This phenomenon can be explained with the fact 
that bisimilarity compares the observable behavior of two states in a system 
and the Markov chains are very simple systems in which there is not much to 
observe. Therefore the need comes to enrich Markov chains with actions or at 
least termination. 
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Notation. In Section 3 we will introduce ten other types of probabilistic au- 
tomata, with corresponding notions of bisimulation. In order to avoid repetition 
we collect the following. 

— A type of automata will always be a triple (. S , A, a) where S' is a set of states, 
A is a set of actions and a is a transition function. The difference between 
the system types is expressed with the difference in the codomains of the 
corresponding transition functions. 

— A bisimulation relation will always be defined as an equivalence on the set 
of states of a system. Depending on the type of systems the “transfer condi- 
tions” in the definition of bisimulation vary. 

— For a particular type of system, the bisimilarity relation, denoted by ss is 
defined by: s « t if and only if there exists a bisimulation R that relates s 
and t, i.e. (s, t) £ R. Although we use the same notation « for bisimilarity 
in different types of systems, it should be clear that for each type of systems, 
« is a different relation. 



2.3 Parallel Composition of LTSs and MCs 

Compositional operators serve the need of modular specification and verification 
of systems. They arise from process calculi, such as CCS ([66]), CSP ([47]) and 
ACP ([19]), where process terms (models of processes) are built from atomic pro- 
cess terms with the use of compositional operators. Usually a model of a process 
calculi is a suitable class of transition systems. Therefore it is often the case that 
process terms are identified with their corresponding transition systems, and the 
compositional operators of the process calculus can be considered as operators 
for combining transition systems. In this overview we focus on the parallel com- 
position operator. The definition of parallel composition varies a lot throughout 
different process calculi. In this section we consider the non-probabilistic case 
(LTSs) in order to explain variants of different parallel compositions, and the 
parallel composition of Markov chains in order to present the basics of proba- 
bilistic parallel composition. 



Labelled Transition Systems. A major distinction between different paral- 
lel composition operators is whether they are synchronous , where the compo- 
nents are forced to synchronize whenever they can, or asynchronous where the 
components can either synchronize or act independently. Furthermore, differ- 
ent approaches for synchronization exist. The result of the parallel composi- 
tion of two automata Ai = (Si, A, aq) and A2 = (S2,A, 02) is an automaton 
A1IIA2 = (Si x S2,A, a) where the definition of a varies. Instead of a pair 
(s, t) £ S 1 x S2 we will write s||f for a state in the composed automaton. Through- 
out this subsection we will use as running example, the parallel composition of 
the following two automata. 




Probabilistic Automata 



9 




CCS style: The set of actions in this case contains compatible actions a, a € A 
and a special idle or internal action r £ A. If one of the automata in state s 
can perform an action a changing to a state s' and the other one in state t 
can perform a’s compatible action a moving to state t! then the composite 
automaton in state s||f can perform the idle action r and move to state s'\\t'. 
Furthermore, independent behavior of each of the automata is possible within 
the composed automaton. 



s\\t A s'\\t' if and only if 

1. s s' , t t',a = t, for b 
and b compatible actions, or 

2. s A s' and if = t, or 

3. t A t' and s' = s. 



a 




a 



The presented CCS parallel composition is asynchronous. A synchronous variant 
(SCCS [65]) is defined by omitting clauses 2. and 3. in the definition above. 



CSP style: Communication or synchronization in a CSP style parallel compo- 
sition occurs on a set of synchronizing actions. Thus actions that are intended 
to synchronize are listed in a set L C A and the rest of the actions can be 
performed independently. 









L = {a} 


So 


1*0 




s\\ L t A 


s' 1 1' if and only if 




a 






1. 


■ s' and t. 


^ t' and a £ L, or 


soPi 


\ 

Sol 


o 

/ 

CN 


1*3 


2. s — > 


■ s', t = t' 


and a ^ T, or 








3. tA 


t' , s = s' 


and a ^ L. 








a 



Y 



S0p4 

This type of parallel composition operator is synchronous for L = A, expresses 
only interleaving (shuffling) composition if L = 0 and is never fully asynchronous 
with both independent behavior and communication allowed. An asynchronous 
CSP style parallel composition can be defined by omitting the clause “a ^ L” 
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in clauses 2. and 3. above. In case of different action sets A\ and A 2 , of the two 
component automata, L is taken to be a subset of Ai D A 2 . If L = Ai n A 2 then 
we say that synchronization on common actions occurs. 



ACP style: In ACP, parallel composition is fully asynchronous, allowing both 
interleaving (independent behavior) and synchronization via a communication 
function. A communication function is a commutative and associative partial 
function 7 : A x A A. Instead of 7 (a, b ) we will write ab. 



s||t A s'\\t' if and only if 

1 . s — > s' and t t! with 
be = a defined, or 

2 . s — > s' , t = t', or 

3. s = s'. 



a 




Note that if A contains compatible actions and an idle action r, and if aa = t 
for any compatible a,a € A and undefined otherwise, then the ACP parallel 
composition operator specializes to the CCS parallel composition operator. On 
the other hand, for aa = a, (a € L C A) we get the asynchronous variant of 
the CSP parallel composition operator. If clauses 2. and 3. are dropped from 
the definition, we get a synchronous variant of the ACP parallel composition 
operator called communication merge. 

Markov Chains. Let Adi = (Si, ai), Ad 2 = ( S 2 ,a 2 ) be two Markov chains. 
Their parallel product is the Markov chain Adi||Ad 2 = (Si x S 2 ,a ), where 
a(s||f) = ai(s) x a 2 (t), x denoting the product of distributions. Hence s\\t fi 
if and only if s Hi, t ^ fi 2 and fi = Hi x fi 2 . 




si s 2 





S0P0 



| N 

Sl 1)^2 S2 ||^2 



Note that the parallel composition of two Markov chains is synchronous, since 
each step in the composed automaton consists of independent steps performed 
by each of the components. The way of defining the product of two distribu- 
tions goes in favor of the interpretation that when put in parallel, each of the 
automata independently chooses its transition that contributes to a transition 
in the composed automaton. 
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3 Probabilistic Models 

This section defines the advanced types of probabilistic automata. The automata 
types are grouped in several subsections reflecting their common properties. 
Basically, every type of probabilistic automata arises from the plain definition of 
a transition system with or without labels. Probabilities can then be added either 
to every transition, or to transitions labelled with the same action, or there can 
be a distinction between probabilistic and ordinary (non-deterministic) states, 
where only the former ones include probabilistic information, or the transition 
function can be equipped with structure that provides both non-determinism 
and probability distributions. 

Each kind of probabilistic automata comes equipped with a notion of bisimu- 
lation, and all these notions, frequently only subtly different, will also find their 
way in this section. 



3.1 Reactive, Generative and I/O Probabilistic Automata 

Two classical extensions of LTSs with probabilities are the reactive and the 
generative model. Throughout the years a large amount of research has been 
devoted to reactive and generative probabilistic systems. It is hard to note who 
introduced these systems first, but the reactive model was treated e.g. in [61, 
62,40,39], the generative in e.g. [40,39,42,50,30,29,28], and the classification 
of these systems together with a so-called stratified model was proposed in [39, 
40]. 

The way these models arise from LTSs, by changing the transition function, 
can be explained with the following figure, where a denotes the transition func- 
tion of a LTS, a r and a g the transition function of a reactive and a generative 
system, respectively. 



[a r : S -+ (V(S) + l)- 4 )^ 



V — » T> + 1 



a:S^V(Sy 



a:S-> V{A x S ) 



v — >c + l 



i^cx g : S -> V{A xg) + l) 



Definition 10. A reactive probabilistic automaton is a triple (S, A, a) where the 
transition function is given by 

a : S —> (V(S) + 1) A . 

If s £ S and a(s)(a) = p a then we write s 
s' € spt(p a ), Ma(s') = p we write s s' . 



p a . More specifically, if 
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A generative probabilistic automaton is a triple (S, A, a) with a transition 
function 

a:S -► V(A x S) + 1. 

When s £ S and a(s) = p £ V(A x 5 1 ) f/ien we write s /z. More particularly, 

if (a, s') £ spt(p) with p((a,s'}) = p we write s s'. We use s '/* to denote 
that a(s) = *. 



Remark 2. In Definition 10 both uses of the special singleton set 1 appear. The 
first one, as in Definition 5 helps expressing partial functions. The second one, 
in the definition of generative transition function, expresses the possibility of 
termination. If s is a state in a generative system with a(s) = * then s is a 
terminating state allowing no transition. For LTSs, termination is allowed by 
the fact that 0 £ V(A x S). Hence, when changing from subsets to distributions, 
* is added to play the role of the 0. 




• • 

Reactive system 



• • 

Generative system 



In a reactive system probabilities are distributed over the outgoing transi- 
tions labelled with the same action, while in a generative system probabilities 
are distributed over all outgoing transitions from a state. A motivation for mak- 
ing this distinction is the different treatment of actions. In a reactive system 
actions are treated as input actions being provided by the environment. When 
a reactive system receives input from the environment then it acts probabilisti- 
cally by choosing the next state according to a probability distribution assigned 
to this input. There are no probabilistic assumptions about the behavior of the 
environment. On the other hand, in a generative system, as the name suggests, 
actions are treated as output generated by the system. When a generative sys- 
tem is in a state s it chooses the next transition according to the probability 
distribution a(s) assigned to s. The transition being chosen, the system moves 
to another state while generating the output action which labels this transition. 
Note that in a generative system there is no non-determinism present, while in a 
reactive system there is only external non-determinism, as in DA. We denote by 
React and Gen the classes of reactive and generative probabilistic automata, 
respectively. 

Definition 11. An equivalence relation R on S is a bisimulation on the reactive 
probabilistic automaton ( S,A,a ) if for all ( s,t ) £ R and for all actions a £ A: 

if s p then there exists a distribution p with t p and p =r p . 
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In order to state the definition of bisimulation for generative systems, the 
lifting from Definition 2 is used. 

Definition 12. An equivalence relation R on S is a bisimulation on the gener- 
ative probabilistic automaton ( S,A,a } if for all ( s,t ) £ R: 

if s^p then there exists a distribution p' with t p' and p =r,a p' ■ 

Example 3. The equivalence relation R generated by the pairs ( C,D ), (if, 1), 
(if, 3), (if, 5), (T, 2), (T, 4), (T, 6) is a bisimulation for the probabilistic automa- 
ton given below. Hence, C ss D. Note that this particular automaton belongs to 
both React and Gen. 



•c 9 D 




An intuitive interpretation of this example is obtained by adding meaning “flip” 
to the action a in the left sub-automaton and a meaning “roll” to the action a 
in the right sub-automaton. Then the state C represents flipping of a fair coin, 
and the state D represents rolling a fair dice. The bisimilarity of the states C 
and D shows that it is the same whether one flips a fair coin or rolls a fair dice 
being interested only in whether the outcome is odd or even. 



I/O Probabilistic Automata. The model of input/output probabilistic au- 
tomata, introduced by Wu, Smolka and Stark in [87], exploiting the input/output 
automata by Lynch and Tuttle, (cf. [63]), presents a combination of the reactive 
and the generative model. 

Definition 13. An input/output probabilistic automaton is a triple (S,A,a) 
where 

1. the set of actions A is divided into input and output actions, A = A m + A out ; 

2. a:S -► V{S) Ain x {V(A out x S) + 1) x is the transition function. 

The third component in the transition function assigns an output delay rate to 
each state. If s € S, then a(s) = (/*", p out , 6 S ). We have that S s = 0 iff p out = * 
i.e. delay is assigned only to states that generate output. 

Denote the class of I/O automata by IO. We use a similar notation for 
transitions as in the reactive and the generative model. If s £ S with ct(s) = 
(. f in ,p out ,6 s ) then 

— if a £ A ln with f m (a) = p a we write s p af furthermore, if s' £ spt(p a ) 
with p a (s') = p we write s s'. 

— if p out ^ * we write s ^ p out and if p out (a, s') = p > 0 we write s s' . 
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transitions from a state in an I/O probabilistic automaton 
A in = {a, b}, A out = {c,d} 

In an I/O automaton for every input action there is a reactive transition. 
Note that f ln is always a function and not a partial function as in the reactive 
model. Hence each input action is enabled in each state of an I/O probabilistic 
automaton. The output actions are treated generatively. At most one generative 
probabilistic transition gives the output behavior of each state. The delay rate 
parameter 6 a is an aspect from continuous-time systems, and its meaning will 
become clear in section 4 when we discuss compositions of I/O automata. 

The I/O automata will not be compared and placed in the hierarchy of sec- 
tion 5 since they involve a continuous element. It is obvious that, when ignoring 
the 0 delays, for A out = 0 one gets the reactive model (with all actions enabled) 
and for A m = 0 one gets the generative model with a delay rate assigned to 
each state. A connection exists between I/O automata and some models with 
structured transition relation (section 3.3). Combined systems similar to I/O au- 
tomata appear as models of process terms in the process algebra EMPA [14, 15]. 

Since we do not compare I/O automata in Section 4, we do not need a notion 
of bisimulation for them, although it can be defined by combining the transfer 
conditions for reactive and generative bisimulation, and taking care of the delay 
rate. In [87] no notion of bisimulation is introduced, instead a different notion 
of behavior of I/O automata is considered. A definition of bisumulation for I/O 
automata can be found in [75]. 

3.2 Automata with Distinction Between States 

So far we have seen some types of automata that allow modelling of proba- 
bilistic behavior, but none of those has the capability of also modelling full 
non-determinism. The types of systems introduced in a minute allow full non- 
determinism while making a distinction between probabilistic states with out- 
going probabilistic transitions, and non-deterministic states with action labelled 
transitions. 



Stratified Probabilistic Automata. The simplest system with a distinction 
on states appears under the name of stratified probabilistic automaton, and is 
discussed in [39,40,79,49]. Stratified automata do not yet allow any form of 
non-determinism although there is a distinction on states. 

Definition 14. A stratified probabilistic automaton is a triple (S', A , a) where 
the transition function a is given by 

a : S — > V{S) + (A x S) + 1 
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The class of all stratified automata we denote by Str. Due to the disjoint 
union in the codomain of the transition function, there are three types of states 
in a stratified automaton: probabilistic states consisting of s G S' such that 
a(s) € T>(S), deterministic states s € S for which a(s) = (a, s') allowing a single 
action labelled transition and terminating states s G S with a(s) = *. 

Definition 15. An equivalence relation R on S is a bisimulation on the strati- 
fied probabilistic automaton ( S,A,a ) if for all ( s,t ) G R: 

1. if s^p then there exists a distribution p' with t~^> p! and p =r p! ; 

2. if s — > s' then there exists t 1 such that t — > t' and (s', t') G R. 

Vardi Probabilistic Automata. One of the earliest models of probabilistic 
automata was introduced by Vardi in [84] under the name concurrent Markov 
chains. The original definition of a concurrent Markov chain was given in terms of 
state labelled transition systems, for purposes of verification of logical properties. 
Therefore we slightly modify the definition, calling this class of automata Vardi 
probabilistic automata. 

Definition 16. A Vardi probabilistic automaton is a triple (S,A,a) where the 
transition function a is given by 

a:S-+ V(A x S) U V(A x S) 



o 




• • • • 



Vardi probabilistic automaton 

Remark 3. Note that U is used in Definition 16 rather than +. One could consider 
the union disjoint, but it is of more use to identify p^ a s ,y with the singleton 

{(a, s')}, i.e. a state with a transition s s' can be identified with a state 
allowing only one transition s s' . 

In Vardi automata, the probabilistic states are of a generative kind, while 
the other states are non-deterministic with full non-determinism, as in an LTS. 
Therefore, the definition of bisimulation is a combination of Definition 8 and 
Definition 12. 

Definition 17. An equivalence relation R on S is a bisimulation on the Vardi 
probabilistic automaton (S,A,a) if for all ( s,t ) G R: 

1. if s^p then there exists a distribution p' with t~~>p' and p =r,a p' ; 

2. if s —>■ s' then there exists t' such that t t' and (s ' , t') G R. 
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Remark 4- We note that in the literature, in particular in [84], there is no defini- 
tion of bisimulation. However, the current understanding of probabilistic bisimu- 
lation, and the concept of a general coalgebraic definition of bisimulation allows 
us to state the previous definition. 

We denote the class of Vardi probabilistic automata by Var. 



The Alternating Models of Hansson. Another model that treats separately 
(purely) probabilistic and non-deterministic states is the alternating model in- 
troduced by Hansson, see for example [41, 46]. We present the class of alternating 
probabilistic automata Alt, its subclass of strictly alternating probabilistic au- 
tomata SA and, in turn, two subclasses of SA, denoted by SA„ and SA p . 

Definition 18 . An alternating probabilistic automaton is a triple (S,A,a) 
where 

a: S ^V(S) + V(Ax S). 

The class of alternating automata is denoted by Alt. Denote by N and P the 
subsets of S containing non-deterministic and probabilistic states, respectively. 

A strictly alternating automaton is an alternating automaton where for all s £ S 
the following holds: 

1. if s £ P with a(s) = p £ V{S) then spt(/.i) C N; 

2. if s £ N then for all (a, s') £ a(s), s' £ P. 

The class of all strictly alternating automata is denoted by SA. 

An automaton of SA belongs to SA„ if and only if 

Vs £ S: (Vs' £ S, Va £ A,Vp £ [0,l]:s'|sAs'/»s)=>s£iV. (1) 

An automaton of SA belongs to SA p if and only if 

Vs £ S’: (Vs' £ S, Va £ A,Vp £ [0, 1] : s' s A s' s) => s £ P. (2) 

The well known of these classes are the class SA [41,46] and the class SA n 
[5,6], but we have chosen for presenting all these classes structurally. The class 
Alt is a slight generalization of the class SA and is very much similar to the 
stratified and Vardi models. Therefore it deserves its place in this overview. 
In an alternating automaton only a distinction on states is imposed. In the 
strictly alternating model it is required that all successors of a non-deterministic 
state are probabilistic states and vice versa. Furthermore, the two subclasses 
SA n and SA p take care that any “initial state” is non-deterministic (1) and 
probabilistic (2), respectively. We define the subclasses SA n and SA p in order 
to make a precise comparison of the class SA with some of the other models 
(section 5). 
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alternating probabilistic automaton strictly alternating automaton (SA n ) 

One definition of bisimulation fits all the introduced classes of alternating 
automata, where the transfer conditions are exactly the same as for the stratified 
model, given in Definition 15. 



3.3 Probabilistic Automata with Structured Transition Function 

In this subsection we focus on three types of probabilistic automata that pro- 
vide orthogonal coexistence of full non-determinism and probabilities without 
distinguishing between states. 



Segala and Simple Segala Probabilistic Automata. Two types of proba- 
bilistic automata were introduced by Segala and Lynch in [78, 76]. We call them 
Segala probabilistic automata and simple Segala probabilistic automata. An ex- 
tensive overview of the simple Segala model is given in [80,81] and they have 
been used for verification purposes and developing theoretical results in several 
situations as reported in [82, 83, 24, 8, 13, 21, 20, 55, 56]. 

Definition 19. A Segala probabilistic automaton is a triple ( S,A,a ) where 

a : S —> V(V(A x S)) 

If s £ S such that p £ a(s) we write s — p, and if (a, s') £ spt(p ) with 
p(a, s') = p then we write s — s' . 

A simple Segala probabilistic automaton 1 is a triple {S, A , a) for a transition 
function 

a: S -4?(4xD(S)) 

If s £ S with ( a,p ) £ a(s) then we write s p, and if s' £ spt(p) we write 

a p , 

s — s . 

The simple Segala type of systems arise from NA by changing the target 
state with a distribution over possible target states. A transition in a simple 
Segala automaton and in a Segala automaton is shown in the next figure. 



1 Segala and Lynch call these models probabilistic automaton (PA) and simple prob- 
abilistic automaton, while Stoelinga calls them general PA and PA, respectively. 
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simple Segala transition Segala transition 

There can be more then one transition available in a state and that is 
where non-determinism occurs. Hence, the non-deterministic choices exist be- 
tween transitions, while the probabilities are specified within a transition. In 
the original definition by Segala and Lynch distributions over an extended set 
AxS + 1 (or over 5 + 1 in the simple case) were treated i.e. substochastic distri- 
butions, where the probability assigned to the special symbol * was interpreted 
as the deadlock probability. We choose not to include this in the definition for 
two reasons: it disturbs the comparison (Section 4) since the other models do not 
include substoclrastic distributions, and it can be imitated by adding an extra 
deadlock state to a system. 

We denote the class of Segala probabilistic automata by Seg and the class 
of simple Segala automata by SSeg. 

The simple Segala automaton is a generalization towards full non- 
determinism of the reactive model and of the purely probabilistic automata of 
Rabin [73]. A deterministic version of the simple Segala automaton equivalent to 
the reactive model is known as Markov decision process ([34]), while the name 
probabilistic transition system is used for this model in [52] and for a state la- 
belled version in [36, 37]. A comparison of SA„ and the simple Segala model can 
be found in [25]. 

Bisimulation for the simple Segala systems is defined with the same transfer 
conditions as for reactive systems given in Definition 11, while for the Segala 
systems the transfer conditions for bisimulation of Definition 12 for generative 
systems apply, when changing to — 

A great novelty introduced with both types of Segala systems was the defi- 
nition of a stronger probabilistic bisimulation relation that identifies states that 
have matching “combined transitions”. For more information on this topic the 
interested reader is referred to [78, 76, 80, 81, 24]. 

Bundle Probabilistic Automata. Another way to include both non- 
determinism and probability is to consider distributions over sets of transitions 
as in the bundle model, introduced in [35]. (Recall that Segala systems have sets 
of distributions over transitions.) 

Definition 20. A bundle probabilistic automaton is a triple (S,A,a) where 

a:S-> V{V(A x S)) + 1 

When s £ S and a(s) = p we write s ^ p, furthermore, if T C A x S, p(T) = 
p > 0 we write s^T and if (a, t) £ T then s t. 




Probabilistic Automata 



19 



The bundle model can be considered as generative, since probabilities are 
also distributed over actions. Therefore the bundle model offers a solution to 
the absence of non-determinism in the generative setting. Note that the original 
definition is even slightly more general, namely the codomain of the transition 
function is D(A4(A x S)) where Xi(X) denotes all the (finite) multi-subsets 
of a set X. Hence it is possible to have multiple transitions from one state to 
another with the same action within one bundle. Since it is not essential for 
the material presented here, we will not add multi-sets in the bundle model. 
The class of bundle probabilistic automata is denoted by Bun. A typical bundle 
probabilistic automaton is depicted below: 




• • • • • 

bundle probabilistic automaton 

In the literature, in particular in [35], there is no definition of bisimulation 
on bundle probabilistic automata, instead they are transformed to generative 
systems and then compared with generative bisimulation. We give here a defini- 
tion of bisimulation for the bundle probabilistic automata that is deduced from 
the general coalgebraic definition of bisimulation (cf. [53, 74, 68]). A justification 
for doing so is that all previously stated definitions of bisimulation which were 
based on the probabilistic bisimulation of Larsen and Skou [61] coincide with the 
general coalgebraic definition of bisimulation for the particular type of systems. 
In the non-probabilistic case this coincidence is well known (see e.g. [74]). For 
Markov chains it was proven in [86] , for the Segala probabilistic automata in [26] 
and the same proof technique extends to all other cases. 

Prior to stating the definition we need a way to lift a relation on a set S to 
a relation on the set V(A x S). 

Definition 21. Let R be a relation on S and let X , Y G V {A x S ) . Define 
X = R <p Y if and only if for all a G A: 

1. if (a,x) G X then there exists (a,y) G Y with (x, y) € R; 

2. if {a,y} G Y then there exists (a,x) G X with (x,y) € R. 

It holds that, if R is an equivalence on S, then = r t> is an equivalence on 
V{A x S). 

Definition 22. An equivalence relation R is a bisimulation on the state set of 
a bundle probabilistic automaton ( S,A,a ) if for all { s,t ) G R it holds 

if s ^ /i then there exists p' such that t p and p == RV p 

where == R V denotes the lifting of the relation =r,v to distributions on V(Ax S) 
as defined by Definition 1. 
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3.4 Complex Models — Pnueli-Zuck and General Probabilistic 
Automata 

An early model including probabilities and a structured transition relation was 
proposed by Pnueli and Zuck [71, 72] under the name finite-state probabilistic 
programs and later used in [7]. We call this type of automata Pnueli-Zuck prob- 
abilistic automata, and denote the class of all such by PZ 2 . The model of Pnueli 
and Zuck has the most complex transition function, it adds one more power set 
to the bundle model and so allows two types of non-determinism, both between 
the probabilistic transitions and inside the transitions. However, in order to get 
a top element for our hierarchy (section 5) we expand the model a bit further 
and define one most general type of probabilistic automata. The class of such 
will be denoted by MG. 

Definition 23. A Pnueli-Zuck automaton is a triple (S, A, a) where 

a : S -► V{V{V{A x 5))) 

When s £ S and p £ a(s) we write s p, further on, if T C A x S, p(T) = 
p> 0 we write s T and if (a, t) £ T then s t. A general probabilistic 

automaton is a triple (S, A , a) where 

a : S -> T(V{V(A x S + S))) 

The notation for Pnueli-Zuck automata is also used for general automata. Fur- 
thermore, if s £ S, p £ a(s),T C A x S + S with p(T) = p > 0 and t £ T , then 
we write s — — » t. 




• •• ••• ••• ••• 

Pnueli-Zuck system most general system 



The unlabelled transitions which appear in the right figure (most general sys- 
tem) correspond to pure probabilistic transitions in Markov chains or alternating 
systems, where a change of state can happen with certain probability without 
performing an action. 

As for bundle systems, there is no notion of bisimulation for Pnueli-Zuck 
systems in the literature. A bisimulation definition can be formulated out of the 



2 Like Vardi’s model, these automata appear in the literature in a state labelled version 
for model checking purposes. Therefore we change the definition towards transition 
labels. 
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general coalgebraic definition, and it leads the same transfer conditions as in 
Definition 22 when changing ^ to — A small modification is needed for the 
general probabilistic automata. 



4 Composing Probabilistic Systems in Parallel 

Having introduced the probabilistic models, we consider possible definitions of 
the parallel composition operator for these extended systems. Lots of results on 
this topic exist in the literature. For a broad overview the reader is referred to 
[35, 9]. An overview of probabilistic process algebras covering other probabilistic 
operators as well is presented in [58] . 

For the classes MC and IO there is a unique parallel composition defined. 
In MC this operation is purely synchronous given by the product of distribu- 
tions (cf. Section 2.3), whereas in IO the definition of the parallel composition 
operation strongly relies on the specific structure of the systems (cf. Section 4.3 
below). For all other classes it is meaningful to consider various definitions of 
parallel composition. Such operations might be synchronous or asynchronous in 
nature and moreover might be based upon the styles CCS, CSP and ACP de- 
scribed in Section 2. The style CSP plays a special role in this respect since it 
is by its definition partly synchronous and partly asynchronous and hence gives 
rise to a somehow mixed variant of parallel composition. 

The classes of (probabilistic) systems can be divided into three groups depen- 
dent on whether they show reactive, generative or alternating behavior. Classes 
belonging to the same of these groups allow in essence similar definition and 
investigation of parallel composition. 

Instead of going through all, obviously quite numerous, variants of parallel 
composition for each single class of systems, we shall in the subsequent sections 
4.1, 4.2 and 4.4. discuss a couple of instructive cases in some detail. However, 
let us give a complete scheme of possible (and/or already studied) definitions of 
parallel composition operator by means of a comprehensive table. In the table 
below each column is dedicated to one class of probabilistic automata, and each 
row to one of the introduced styles of parallel composition. In the intersecting 
cells a symbol representing the definability status of the corresponding parallel 
composition operator in the corresponding class is placed. Neighboring cells con- 
taining the same symbol within one column are merged. We use the following 
symbols: 



+ : defined in the literature or straightforward 

+ : definable but not carried out 

— : not definable 

p: defined in the literature with parameters 

p: parameterized version definable, but not carried out 

n: normalized version definable, but not carried out 

+/n: “+” for total communication function, “n” otherwise 
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Table 1 presents the overall situation concerning definability of parallel com- 
position on probabilistic automata. A brief analysis of these summary results 
shows that allowing full non-determinism enables definition of any type of par- 
allel composition. 



Table 1. Definability of 
generative 



reactive alternating 
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4.1 Parallel Composition in the Reactive Setting 

Systems with reactive behavior are systems in the classes React and SSeg, 
as well as NA and DA in the non-probabilistic case. Any parallel compo- 
sition operator on LTS (Section 2.3.) nicely extends to the class SSeg. Let 
A\ = (Si, A, ai), A-2 = (S2, A, 02) be in SSeg. Then A1IIA2 = (Si x S2,A,a) 
where a is defined as follows: 

[CCS style}: s||f h if and only if 

1. a = r, s hi, t H2 and h = Hi x H 2 , or 

2. s hi and /i = hi x Mt> or 

3. t A~»- n 2 anc l n = x H2- 

[CSP style]: s||if h if and only if 

1 . a £ L, s Hit t H2 and H = L L 1 x M2> or 

2. a L, s p ( 1 and h = Hi x Mt > or 

3. a (ji L, t pi 2 and h = hI x M 2 - 

[ACP style]: s\\t pi if and only if 

Jj C 

1 . a = be defined, s — pii, t — pi 2 and /r = /k 1 x H2, or 

2. s p L± and h = Hi x Mt > or 

3. t pi 2 and h — Ms x M 2 - 
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The definition of any of these operators is problematic for the class React. 
For Ai,A 2 € React it might happen that ^4i 11^4.2 ^ React in any variant of 
parallel composition. Even in the synchronous CCS style, multiple transitions 
labelled with r may appear. In the CSP style, 2. and 3. may introduce internal 
non-determinism. However, if L contains all the common actions of „4i and 
A 2 , then this problem disappears. In case of ACP all of 1., 2. and 3. introduce 
internal non-determinism, hence React is not closed under this operator for 
any arbitrary communication function y 3 . For example, if ab = ac = a then the 
ACP parallel product of the following two automata 



X' 

1 1 1 2 t 3 

is not defined in React, since the definition yields: s 0 \\t 0 A for a: € 

{si||£oi So||t2> si||fi, S1P3} i.e. more than one transition corresponds to the ac- 
tion a, which is prohibited in React. 

An asynchronous parallel composition in CCS style on simple Segala systems 
was defined in [21], a synchronous parallel composition in CCS style on reac- 
tive systems was defined in [40,39,52], the last reference working with simple 
Segala systems. A synchronous CSP style parallel composition is defined for re- 
active systems in [55,69], while an asynchronous CSP style parallel composition 
with synchronization on common actions is used in [78,76,80] for simple Segala 
systems. 

4.2 Parallel Composition in the Generative Setting 

Systems with generative behavior belong to the classes Gen, Var, Seg, Bun, 
PZ and MG. The Vardi systems express also alternating behavior and they will 
be discussed with the alternating systems. A common property of the generative 
systems is that always probability distributions over actions and states appear. 
This leads to difficulties in defining parallel composition operators (see [41,30, 
76,35]), especially in the asynchronous case. Namely, a generative type system 
defines in each state a probability distribution over a set of enabled actions, 
offered by the environment. When two such systems are composed in parallel it 
is not clear how the common set of enabled actions should be defined, nor how 
the two probability distributions should be composed into one (cf. [52]). In this 
section we explain several approaches for solving this problem. 

Let Ai = (Si, A, a\),A 2 = (S 2 ,A,a 2) be two generative systems. Their 
parallel composition in all cases will be denoted by _4i ||^4 2 = (Si x 52, A, a), 
possibly with parameters. 



3 The same problems arise in the class DA opposed to NA, namely parallel compo- 
sition introduces internal non-determinism, and therefore DA is not closed under 





